1. Field of the Invention
The present invention generally relates to neural networks and particularly to outer product neural networks used to extract principal components from any type of signal.
2. Statement of the Prior Art
The term "neural network" is used here to describe any conglomeration of parallel, identical processor elements (PEs) whether linear or non-linear which may be jointly trained to achieve some state or minimal condition based on minimizing a constraint. The term "principal components" is being used loosely here as the major energy modes of the signal of interest. The signal of interest may be any one to n-dimensional signal such as a time series, image or n-dimensional feature vectors.
Beginning with Oja (E. Oja, "A Simplified Neuron Model As A Principal Component Analyzer": J. Math. Biology, vol. 15, pp. 267-273, 1982), it was recognized that neural networks could be used to extract principal components from a signal. Formally, principal components are the axes of an n-dimensional Gaussian signal; here, the term is used somewhat loosely to describe the principal modes and components of a signal. If most of the signal energy may be represented by a few principal components or modes, then the signal may be `explained` by those few principal components resulting in a reduced-rank representation of the signal. For instance, the reconstituted signal derived from the principal components may be used in place of the original signal and the principal components may be used for signal transmission and storage representing the original signal in compressed form. This will also, in general gain some advantage such as eliminating noise or enhancing the signal for subsequent processing based on signal rank.
Oja's rule was sufficient for extracting the single, largest principal component from a signal but could not extract multiple principal components. Sanger (T. Sanger, "Optimal Unsupervised Learning In A Single Layer Feed Forward Neural Network", presented at NIPS, Denver, Colo., Nov. 29-Dec. 1, 1988, pp 1-17) revised Oja's network and rule to enable extraction of multiple principal components. Sanger's rule required interconnection among the processor elements to enforce orthogonality. This is required as the principal component must form an orthonormal basis for the principal component subspace. Sanger also required a second normalization step.
More recently, Foldiak (P. Foldiak, "Adaptive Network For Optimal Linear Feature Extraction", Proc. IJCNN, pp. 1401-1406, Washington, D.C., 1989) and Kung (S. Y. Kung and K. I. Diamantaras, "A Neural Network Learning Algorithm For Adaptive Principal Components Extraction (APEX)", in Proc. ICASSP, vol. 2, pp. 861-864, 1990) have devised neural networks for extracting principal components. The updating rules for these networks also have interconnections among the processor elements to enforce the `anti-Hebbian` training, i.e. to constrain the principal components to be orthonormal. Of course, these additional connections require more processing be done for the network to converge.
Oja (E. Oja, "Neural Networks, Principal Components, and Subspaces", Int. J. Neural Systems, Vol. 1, No. 1, pp. 61-68, 1989) extended his original seminal paper based on a different line of reasoning which rendered his network very sensitive, and somewhat unstable, to training parameters.